oasis

2022-04-14 ยท 13 min read

Site: https://oasisprotocol.org/

Github: https://github.com/oasisprotocol/, oasis-sdk, oasis-core

Lots of interesting/useful SGX-related stuff

Oasis Run a Node Process (Setting up SGX)

Run an Intel Attestation Service (IAS) proxy

TODO: enclave rpc

Core Oasis Runtime Model #

Oasis Core Compute Nodes run user executables inside SGX enclaves.

               ____________________
			  |  ________________  |
              | | Runtime Loader | |
			  | '----------------' |
			  |      Enclave	   |
 ______       |  ________________  |
| Node |<------>|  Runtime Bin   | |
'______'      | '----------------' |
              '--------------------'
  • The Rust oasis-core/runtime is the in-enclave runtime. This is "Runtime Loader" and "Runtime Bin" in the above diagram.
  • The Go oasis-core/go/runtime/host is responsible for provisioning enclaves and then communicating with them. This is part of "Node" in the above diagram.
  • The Node and Runtime communicate via
    • Node <-> Unix Socket <-> Enclave Runner <-> Fortanix shared-memory FIFO queue <-> Enclave Runtime Binary

oasis-core Runtime Loader #

https://github.com/oasisprotocol/oasis-core/tree/master/runtime-loader

Inputs:

  1. Runtime Filename
  2. Signature
  3. host-socket

SgxsLoader

  1. Opens the SGX kernel devices for loading enclaves.
  2. Opens client connection to AESM service for the EINITTOKEN provider.
  3. Build enclave from file, SIGSTRUCT signature, and their custom adapter, which just routes connect syscalls from the enclave to a local Unix Domain Socket (UDS). This connection is the bidirectional arrow between Node and Runtime Bin in the above diagram.
/// SGX runtime loader.
pub struct SgxsLoader;

impl Loader for SgxsLoader {
    fn run(
        &self,
        filename: String,
        signature_filename: Option<&str>,
        host_socket: String,
    ) -> Fallible<()> {
        let sig = match signature_filename {
            Some(f) => f,
            None => {
                return Err(format_err!("signature file is required"));
            }
        };

        // Spawn the SGX enclave.
        let mut device = IsgxDevice::new()?
            .einittoken_provider(AesmClient::new())
            .build();

        let mut enclave_builder = EnclaveBuilder::new(filename.as_ref());
        enclave_builder.signature(sig)?;
        enclave_builder.usercall_extension(HostService::new(host_socket));
        let enclave = enclave_builder.build(&mut device)?;

        enclave.run()
    }
}

oasis-core Host # 

// Manifest is a deserialized runtime bundle manifest.
type Manifest struct {
	// Name is the optional human readable runtime name.
	Name string `json:"name,omitempty"`

	// ID is the runtime ID.
	ID common.Namespace `json:"id"`

	// Version is the runtime version.
	Version version.Version `json:"version,omitempty"`

	// Executable is the name of the runtime ELF executable file.
	Executable string `json:"executable"`

	// SGX is the SGX specific manifest metadata if any.
	SGX *SGXMetadata `json:"sgx,omitempty"`

	// Digests is the cryptographic digests of the bundle contents,
	// excluding the manifest.
	Digests map[string]hash.Hash `json:"digests"`
}

// SGXMetadata is the SGX specific manifest metadata.
type SGXMetadata struct {
	// Executable is the name of the SGX enclave executable file.
	Executable string `json:"executable"`

	// Signature is the name of the SGX enclave signature file.
	Signature string `json:"signature"`
}

// Bundle is a runtime bundle instance.
type Bundle struct {
	Manifest *Manifest
	Data     map[string][]byte
}


// RuntimeBundle is a exploded runtime bundle ready for execution.
type RuntimeBundle struct {
	*bundle.Bundle

	// Exeuctable is the path to the extracted ELF or TEE executable.
	Path string
}

// Config contains common configuration for the provisioned runtime.
type Config struct {
	// Bundle is the runtime bundle.
	Bundle *RuntimeBundle

	// Extra is an optional provisioner-specific configuration.
	Extra interface{}

	// MessageHandler is the message handler for the Runtime Host Protocol messages.
	MessageHandler protocol.Handler

	// LocalConfig is the node-local runtime configuration.
	LocalConfig map[string]interface{}
}

Runtime Host Protocol #

This is the core API layer available for executables running inside an SGX enclave. Nodes communicate bidirectionalyl with the running user binary via a bespoke length-prefixed CBOR message stream protocol.

Connection Lifecycle #
  • Uninitialized - Newly created connection. Must be initialized before use.
  • Inititalizing - Parties exchange version and feature info.
  • Ready - The versions and features are negotiated; however, we first need to perform remote attestation before running stuff in the enclave.
  • Closed - Connection is closed.
Remote Attestation #

Host attestation flow - oasis-core/go/runtime/host/sgx/sgx.go

  1. (Host) get the Quoting Enclave (QE)'s QuoteInfo from the AESM.
  2. (Host) get the Service Provider ID (SPID) from the Intel Attestation Services (IAS) cache.
  3. (Host) send a RuntimeCapabilityTEERakInitRequest { QuoteInfo.TargetInfo } request to the enclave.
  4. (Enclave) receive the request. initialize the Runtime Attestation Key (RAK) with the TargetInfo and then generate a new ephemeral RAK keypair (if one doesn't exist already).
  5. (Host) ask the IAS cache for the latest EPID revocation list (since Oasis uses EPID).
  6. (Host) send a RuntimeCapabilityTEERakReportRequest {} to the enclave.
  7. (Enclave) receive the request.
    1. Sample a random nonce.
    2. Generate report_body = H(RAK_pub). This binds the RAK pubkey to the report.
    3. Set report_data = report_body || nonce.
    4. Generate report = Report::for_target(target_info, report_data) from the enclave EREPORT instruction. See: Report (EREPORT).
    5. Return the RAK public key, report, and nonce to the host.
  8. (Host) Request a Quote from the AESM using the report, SPID, and sigRL revocation list. For some reason the nonce is empty here? Not sure what the nonce here is for...
  9. (Host) Verify that the quote.Report contains a MRENCLAVE and MRSIGNER that we expect. This ensures we're talking to the right application enclave.
  10. (Host) Verify that the Quote.Report.Attributes.is_debug == config.is_debug. That way we don't trust debug enclaves in production and don't use production enclaves in testing.
  11. (Host) Ask the IAS to Verify Evidence { quote, nonce }. IAS will return an Attestation Verification Report (AVR) to us.
  12. (Host) send a RuntimeCapabilityTEERakAvrRequest { avr } request to the enclave, to pass it the AVR.
    1. (Q: why does the enclave need to verify the AVR? shouldn't the client do this? I suppose we already checked that the enclave is the one we expect; in which case, we trust it to verify the AVR correctly.)
  13. (Enclave) receive the AVR in the request.
    1. Expect the nonce in the AVR to match the one we generated above in step (7.1).
    2. See Enclave Attestation Verification Report AVR handling.
    3. Verify the current measured Report's enclave identity matches the enclave identity in the AVR.
    4. Verify that the RAK pubkey matches the one in the AVR.report_data.
    5. Verify that the nonce also matches the one in the AVR.report_data.
    6. Ratchet our internal timestamp from the one we've observed.
  14. (Host) we've now fully verified the enclave. package up the RAK pubkey and AVR into a capabilityTEE.

Other nodes can also verify the capabilityTEE by

  1. checking the AVR against the Intel Trust Root CA
  2. checking the MRENCLAVE/MRSIGNER are what we expect
  3. checking that the AVR.Quote.Report.ReportData includes the expected RAK pubkey
type SPIDInfo struct {
	SPID               ias.SPID
	QuoteSignatureType ias.SignatureType
}

// A deserialized AVRBundle from IAS.
type AttestationVerificationReport struct {
	ID                    string                `json:"id"`
	Timestamp             string                `json:"timestamp"`
	Version               int                   `json:"version"`
	ISVEnclaveQuoteStatus ISVEnclaveQuoteStatus `json:"isvEnclaveQuoteStatus"`
	ISVEnclaveQuoteBody   []byte                `json:"isvEnclaveQuoteBody"`
	RevocationReason      *CRLReason            `json:"revocationReason"`
	PSEManifestStatus     *PSEManifestStatus    `json:"pseManifestStatus"`
	PSEManifestHash       string                `json:"pseManifestHash"`
	PlatformInfoBlob      string                `json:"platformInfoBlob"`
	Nonce                 string                `json:"nonce"`
	EPIDPseudonym         []byte                `json:"epidPseudonym"`
	AdvisoryURL           string                `json:"advisoryURL"`
	AdvisoryIDs           []string              `json:"advisoryIDs"`
}

See: Quoting Enclave's QuoteInfo

Enclave Attestation Verification Report (AVR) handling #

https://github.com/oasisprotocol/oasis-core/blob/master/runtime/src/common/sgx/avr.rs

// AVR signature validation constants.
const IAS_TRUST_ANCHOR_PEM: Vec<u8> = parse_x509_pem(
	r#"-----BEGIN CERTIFICATE-----
	MIIFSzCCA7OgAwIBAgIJANEHdl0yo7CUMA0GCSqGSIb3DQEBCwUAMH4xCzAJBgNV
	..
	DD+gT9sSpssq0ascmvH49MOgjt1yoysLtdCtJW/9FZpoOypaHx0R+mJTLwPXVMrv
	DaVzWh5aiEx+idkSGMnX
	-----END CERTIFICATE-----"#
).unwrap();
const PEM_CERTIFICATE_LABEL: &str = "CERTIFICATE";
const IAS_TS_FMT: &str = "%FT%T%.6f";

/// Attestation verification report.
#[derive(Debug, Clone, cbor::Encode, cbor::Decode)]
pub struct AVR {
	// json blob
	//
	// { "isvEnclaveQuoteStatues": __
	// , "isvEnclaveQuoteBody": __
	// , "timestamp": __
	// , "nonce": __
    // }
    pub body: Vec<u8>,
    pub signature: Vec<u8>,
    pub certificate_chain: Vec<u8>,
}

/// Enclave identity.
#[derive(Debug, Clone, Hash, Eq, PartialEq, cbor::Encode, cbor::Decode)]
pub struct EnclaveIdentity {
    pub mr_enclave: MrEnclave,
    pub mr_signer: MrSigner,
}

/// Authenticated information obtained from validating an AVR.
#[derive(Debug, Clone)]
pub struct AuthenticatedAVR {
    pub report_data: Vec<u8>,
    // TODO: add other av report/quote body/report fields we want to give
	//       the consumer
    pub identity: EnclaveIdentity,
    pub timestamp: i64,
    pub nonce: String,
}

/// Decoded quote body.
#[derive(Default, Debug)]
struct QuoteBody {
    version: u16,
    signature_type: u16,
    gid: u32,
    isv_svn_qe: u16,
    isv_svn_pce: u16,
    basename: [u8; 32],
    report_body: Report,
}

impl AVR {
	pub fn verify(&self) -> Result<AuthenticatedAVR> {
		// 1. read configs if we should skip verification during
		//    fuzzing/testing.
		// 2. validate_avr_signature(self.cert_chain, self.body,
		//    self.sig. time::now())
		// 3. check if timestamp in AVR is too old
		// 4. verify ISV Enclave Quote Status == OK | SW_HARDENING_NEEDED
		//    if lax verification enabled, also allow some other errors
		// 5. base64 decode ISV Enclave Quote Body, then decode to `QuoteBody`
		// 6. Possibly allow debug enclaves if appropriate config.
		// 7. They appear to use the IAS timestamp as a semi-trusted source
		//    and ratchet their local clock forward if the IAS timestamp is
		//    farther ahead.
	}
}

fn validate_avr_signature(
	cert_chain: &[u8],
	message: &[u8],
	signature: &[u8],
	unix_timestamp: u64,
) -> Result<()> {
	// * They note that they're doing some funky process that barely resembles
	//   the standard cert verification process.
	// * Don't do cert revocation checks.
	// * Intel Docs: The AVR is signed by the Report Signing Key using
	//   RSA-SHA256
	// * Intel Docs: The Report Signing pubkey is distributed via an x509
	//   cert. This cert is a leaf cert issued by the "Attestation Report
	//   Signing CA".
	// * A PEM-encoded cert chain is (1) the Report Signing Key Cert and (2)
	//   the Report Signing CA.

	// 1. PEM decode the cert chain to DER. Check that each cert has the
	//    "CERTIFICATE" label.
	// 2. Check that only two certs were returned.
	// 3. Ensure the last cert is the Report Signing CA (IAS_TRUST_ANCHOR).
	// 4. Ensure the CA cert is not expired, is a CA cert, is self-signed.
	// 5. Parse the leaf cert, verify the usual stuff, check CA sig.
	// 6. Pull out the leaf pubkey. Verify `signature` on `message`. The
	//    pubkey should be an RSA PKCS1 DER-encoded pubkey. The sig should be
	//    RSA-SHA256 PKCS1v15.
}
Runtime Attestation Key (RAK) #

RAK is the identity of the enclave, used to sign remote attestations.

/// The runtime attestation key (RAK) represents the identity of the enclave
/// and can be used to sign remote attestations. Its purpose is to avoid
/// round trips to IAS for each verification as the verifier can instead
/// verify the RAK signature and the signature on the provided AVR which
/// RAK to the enclave.
pub struct RAK {
    inner: RwLock<Inner>,
}

struct Inner {
    private_key: Option<PrivateKey>,
    avr: Option<Arc<avr::AVR>>,
    avr_timestamp: Option<i64>,
    #[allow(unused)]
    enclave_identity: Option<avr::EnclaveIdentity>,
    #[allow(unused)]
    target_info: Option<Targetinfo>,
    #[allow(unused)]
    nonce: Option<String>,
}
EGETKEY #

https://github.com/oasisprotocol/oasis-core/blob/master/runtime/src/common/sgx/egetkey.rs

  • KMac: SHA3-256 Keccak MAC
/// egetkey returns a 256 bit key suitable for sealing secrets to the
/// enclave in cold storage, derived from the results of the `EGETKEY`
/// instruction.  The `context` field is a domain separation tag.
///
/// Note: The key can also be used for other things (eg: as an X25519
/// private key).
pub fn egetkey(key_policy: Keypolicy, context: &[u8]) -> [u8; 32] {
    let mut k = [0u8; 32];

    // Obtain the per-CPU package SGX sealing key, with the requested
    // policy.
    let master_secret = egetkey_impl(key_policy, context);

    // Expand the 128 bit EGETKEY result into a 256 bit key, suitable
    // for use with our MRAE primitives.
    let mut kdf = KMac::new_kmac256(
		&master_secret,
		b"Ekiden Expand SGX Seal Key",
	);
    kdf.update(context);
    kdf.finalize(&mut k);

    k
}

/// The SGX impl
fn egetkey_impl(
	// MRSIGNER or MRENCLAVE
	key_policy: Keypolicy,
	// the domain separation value
	context: &[u8],
) -> [u8; 16] {
	// As we can see, using the Keyrequest::default() means oasis-core doesn't
	// bind the key
    let mut req = Keyrequest::default();
    req.keyname = Keyname::Seal as u16;
    req.keypolicy = key_policy;

    let mut sha3 = Sha3::v256();
    sha3.update(context);
    let mut k = [0; 32];
    sha3.finalize(&mut k);
    req.keyid = k;

    // Fucking sgx_isa::Attributes doesn't have a -> [u64;2].
	// SGX_FLAGS_INITTED | SGX_FLAGS_DEBUG | SGX_FLAGS_MODE64BIT
    req.attributemask[0] = 1 | 2 | 4;
	// SGX_XFRM_LEGACY
    req.attributemask[1] = 3;

    match req.egetkey() {
        Err(e) => panic!("EGETKEY failed: {:?}", e),
        Ok(k) => k,
    }
}
Sealing #

https://github.com/oasisprotocol/oasis-core/blob/master/runtime/src/common/sgx/seal.rs

  • DeoxysII - Some new symmetric key AEAD cipher that uses AESNI. Not sure why they use this and not just like AES-GCM or AES-SIV. Maybe safer nonce reuse protection?
/// Query the enclave for sealing key material with the current context and
/// key policy, then generate a DeoxysII symmetric key using the key material.
fn new_d2(key_policy: Keypolicy, context: &[u8]) -> DeoxysII {
    let mut seal_key = egetkey(key_policy, context);
    let d2 = DeoxysII::new(&seal_key);
    seal_key.zeroize();

    d2
}

/// Seal a secret to the enclave.
pub fn seal(
	/// MRSIGNER or MRENCLAVE
	key_policy: Keypolicy,
	/// The domain separation value
	context: &[u8],
	/// The plaintext data we want to seal
	data: &[u8],
) -> Vec<u8> {
    let mut rng = OsRng {};

    let mut nonce = [0u8; NONCE_SIZE];
    rng.fill(&mut nonce);
    let d2 = new_d2(key_policy, context);
    let mut ciphertext = d2.seal(&nonce, data.to_vec(), vec![]);
    ciphertext.extend_from_slice(&nonce);

	// ciphertext = [ E_k[data] || MAC tag || nonce ]
    ciphertext
}



/// Unseal a previously sealed secret to the enclave.
///
/// # Panics
///
/// All parsing and authentication errors of the ciphertext are fatal and
/// will result in a panic.
pub fn unseal(
	/// MRSIGNER or MRENCLAVE
	key_policy: Keypolicy,
	/// The domain separation value
	context: &[u8],
	/// Previously sealed data
	ciphertext: &[u8],
) -> Option<Vec<u8>> {
    let ct_len = ciphertext.len();
    if ct_len == 0 {
        return None;
    }
    assert!(
        ct_len >= TAG_SIZE + NONCE_SIZE,
        "ciphertext is corrupted, invalid size"
    );
    let ct_len = ct_len - NONCE_SIZE;

    // split: ciphertext = [ E_k[data] || MAC tag || nonce ]
    let mut nonce = [0u8; NONCE_SIZE];
    nonce.copy_from_slice(&ciphertext[ct_len..]);
    let ciphertext = &ciphertext[..ct_len];

    let d2 = new_d2(key_policy, context);
    let plaintext = d2
        .open(&nonce, ciphertext.to_vec(), vec![])
        .expect("ciphertext is corrupted");

    Some(plaintext)
}
Key Manager Enclaves #

They have some keymanager enclave, which appears to manually handle delegating access to individual enclave seal keys.

  • They use MRENCLAVE for each indiviual enclave (so the key is bound to the machine) then the enclave can register the actual key with the keymanager along with a policy to mediate access more granularly.
  • Their current policy format allows you to explicitly enumerate which enclaves (MRENCLAVE + MRSIGNER) may query private key material and which enclaves may replicate the "master secret" (something consensus related?).